{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example of extracting features from dataframes with Datetime indices\n",
    "\n",
    "Assuming that time-varying measurements are taken at regular intervals can be sufficient for many situations. However, for a large number of tasks it is important to take into account **when** a measurement is made. An example can be healthcare, where the interval between measurements of vital signs contains crucial information. \n",
    "\n",
    "Tsfresh now supports calculator functions that use the index of the timeseries container in order to calculate the features. The only requirements for these function is that the index of the input dataframe is of type `pd.DatetimeIndex`. These functions are contained in the new class TimeBasedFCParameters.\n",
    "\n",
    "Note that the behaviour of all other functions is unaffected. The settings parameter of `extract_features()` can contain both index-dependent functions and 'regular' functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-22T10:19:26.504073Z",
     "start_time": "2019-03-22T10:19:26.397323Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from tsfresh.feature_extraction import extract_features\n",
    "# TimeBasedFCParameters contains all functions that use the Datetime index of the timeseries container\n",
    "from tsfresh.feature_extraction.settings import TimeBasedFCParameters  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build a time series container with Datetime indices\n",
    "\n",
    "Let's build a dataframe with a datetime index. The format must be with a `value` and a `kind` column, since each measurement has its own timestamp - i.e. measurements are not assumed to be simultaneous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-22T10:09:14.175077Z",
     "start_time": "2019-03-22T10:09:14.119751Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>value</th>\n",
       "      <th>kind</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-03-01 10:04:00</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>temperature</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-03-01 10:50:00</th>\n",
       "      <td>a</td>\n",
       "      <td>2</td>\n",
       "      <td>temperature</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-03-02 00:00:00</th>\n",
       "      <td>a</td>\n",
       "      <td>3</td>\n",
       "      <td>pressure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-03-02 09:04:59</th>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>pressure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-03-02 23:54:12</th>\n",
       "      <td>b</td>\n",
       "      <td>3</td>\n",
       "      <td>temperature</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-03-03 08:13:04</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>temperature</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-03-04 08:00:00</th>\n",
       "      <td>b</td>\n",
       "      <td>0</td>\n",
       "      <td>pressure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-03-04 08:01:00</th>\n",
       "      <td>b</td>\n",
       "      <td>8</td>\n",
       "      <td>pressure</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    id  value         kind\n",
       "2019-03-01 10:04:00  a      1  temperature\n",
       "2019-03-01 10:50:00  a      2  temperature\n",
       "2019-03-02 00:00:00  a      3     pressure\n",
       "2019-03-02 09:04:59  a      1     pressure\n",
       "2019-03-02 23:54:12  b      3  temperature\n",
       "2019-03-03 08:13:04  b      1  temperature\n",
       "2019-03-04 08:00:00  b      0     pressure\n",
       "2019-03-04 08:01:00  b      8     pressure"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({\"id\": [\"a\", \"a\", \"a\", \"a\", \"b\", \"b\", \"b\", \"b\"], \n",
    "                   \"value\": [1, 2, 3, 1, 3, 1, 0, 8],\n",
    "                   \"kind\": [\"temperature\", \"temperature\", \"pressure\", \"pressure\",\n",
    "                            \"temperature\", \"temperature\", \"pressure\", \"pressure\"]},\n",
    "                   index=pd.DatetimeIndex(\n",
    "                       ['2019-03-01 10:04:00', '2019-03-01 10:50:00', '2019-03-02 00:00:00', '2019-03-02 09:04:59',\n",
    "                        '2019-03-02 23:54:12', '2019-03-03 08:13:04', '2019-03-04 08:00:00', '2019-03-04 08:01:00']\n",
    "                   ))\n",
    "df = df.sort_index()\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Right now `TimeBasedFCParameters` only contains `linear_trend_timewise`, which performs a calculation of a linear trend, but using the time difference in hours between measurements in order to perform the linear regression. As always, you can add your own functions in `tsfresh/feature_extraction/feature_calculators.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-22T10:09:23.054729Z",
     "start_time": "2019-03-22T10:09:23.006889Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'linear_trend_timewise': [{'attr': 'pvalue'},\n",
       "  {'attr': 'rvalue'},\n",
       "  {'attr': 'intercept'},\n",
       "  {'attr': 'slope'},\n",
       "  {'attr': 'stderr'}]}"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "settings_time = TimeBasedFCParameters()\n",
    "settings_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We extract the features as usual, specifying the column value, kind, and id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-22T10:09:35.217431Z",
     "start_time": "2019-03-22T10:09:35.009287Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Feature Extraction: 100%|██████████| 4/4 [00:00<00:00, 591.10it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>variable</th>\n",
       "      <th>pressure__linear_trend_timewise__attr_\"intercept\"</th>\n",
       "      <th>pressure__linear_trend_timewise__attr_\"pvalue\"</th>\n",
       "      <th>pressure__linear_trend_timewise__attr_\"rvalue\"</th>\n",
       "      <th>pressure__linear_trend_timewise__attr_\"slope\"</th>\n",
       "      <th>pressure__linear_trend_timewise__attr_\"stderr\"</th>\n",
       "      <th>temperature__linear_trend_timewise__attr_\"intercept\"</th>\n",
       "      <th>temperature__linear_trend_timewise__attr_\"pvalue\"</th>\n",
       "      <th>temperature__linear_trend_timewise__attr_\"rvalue\"</th>\n",
       "      <th>temperature__linear_trend_timewise__attr_\"slope\"</th>\n",
       "      <th>temperature__linear_trend_timewise__attr_\"stderr\"</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>-0.22019</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.304348</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>480.00000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>-0.240545</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "variable  pressure__linear_trend_timewise__attr_\"intercept\"  \\\n",
       "id                                                            \n",
       "a                                                       3.0   \n",
       "b                                                       0.0   \n",
       "\n",
       "variable  pressure__linear_trend_timewise__attr_\"pvalue\"  \\\n",
       "id                                                         \n",
       "a                                                    0.0   \n",
       "b                                                    0.0   \n",
       "\n",
       "variable  pressure__linear_trend_timewise__attr_\"rvalue\"  \\\n",
       "id                                                         \n",
       "a                                                   -1.0   \n",
       "b                                                    1.0   \n",
       "\n",
       "variable  pressure__linear_trend_timewise__attr_\"slope\"  \\\n",
       "id                                                        \n",
       "a                                              -0.22019   \n",
       "b                                             480.00000   \n",
       "\n",
       "variable  pressure__linear_trend_timewise__attr_\"stderr\"  \\\n",
       "id                                                         \n",
       "a                                                    0.0   \n",
       "b                                                    0.0   \n",
       "\n",
       "variable  temperature__linear_trend_timewise__attr_\"intercept\"  \\\n",
       "id                                                               \n",
       "a                                                       1.0      \n",
       "b                                                       3.0      \n",
       "\n",
       "variable  temperature__linear_trend_timewise__attr_\"pvalue\"  \\\n",
       "id                                                            \n",
       "a                                                       0.0   \n",
       "b                                                       0.0   \n",
       "\n",
       "variable  temperature__linear_trend_timewise__attr_\"rvalue\"  \\\n",
       "id                                                            \n",
       "a                                                       1.0   \n",
       "b                                                      -1.0   \n",
       "\n",
       "variable  temperature__linear_trend_timewise__attr_\"slope\"  \\\n",
       "id                                                           \n",
       "a                                                 1.304348   \n",
       "b                                                -0.240545   \n",
       "\n",
       "variable  temperature__linear_trend_timewise__attr_\"stderr\"  \n",
       "id                                                           \n",
       "a                                                       0.0  \n",
       "b                                                       0.0  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_tsfresh = extract_features(df, column_id=\"id\", column_value='value', column_kind='kind',\n",
    "                             default_fc_parameters=settings_time)\n",
    "X_tsfresh.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output looks exactly, like usual. If we compare it with the 'regular' `linear_trend` feature calculator, we can see that the intercept, p and R values are the same, as we'd expect – only the slope is now different."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-22T10:10:12.746410Z",
     "start_time": "2019-03-22T10:10:12.717705Z"
    }
   },
   "outputs": [],
   "source": [
    "settings_regular = {'linear_trend': [\n",
    "  {'attr': 'pvalue'},\n",
    "  {'attr': 'rvalue'},\n",
    "  {'attr': 'intercept'},\n",
    "  {'attr': 'slope'},\n",
    "  {'attr': 'stderr'}\n",
    "]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-22T10:10:23.278247Z",
     "start_time": "2019-03-22T10:10:23.100779Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Feature Extraction: 100%|██████████| 4/4 [00:00<00:00, 2517.59it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>variable</th>\n",
       "      <th>pressure__linear_trend__attr_\"intercept\"</th>\n",
       "      <th>pressure__linear_trend__attr_\"pvalue\"</th>\n",
       "      <th>pressure__linear_trend__attr_\"rvalue\"</th>\n",
       "      <th>pressure__linear_trend__attr_\"slope\"</th>\n",
       "      <th>pressure__linear_trend__attr_\"stderr\"</th>\n",
       "      <th>temperature__linear_trend__attr_\"intercept\"</th>\n",
       "      <th>temperature__linear_trend__attr_\"pvalue\"</th>\n",
       "      <th>temperature__linear_trend__attr_\"rvalue\"</th>\n",
       "      <th>temperature__linear_trend__attr_\"slope\"</th>\n",
       "      <th>temperature__linear_trend__attr_\"stderr\"</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>-2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>-2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "variable  pressure__linear_trend__attr_\"intercept\"  \\\n",
       "id                                                   \n",
       "a                                              3.0   \n",
       "b                                              0.0   \n",
       "\n",
       "variable  pressure__linear_trend__attr_\"pvalue\"  \\\n",
       "id                                                \n",
       "a                                           0.0   \n",
       "b                                           0.0   \n",
       "\n",
       "variable  pressure__linear_trend__attr_\"rvalue\"  \\\n",
       "id                                                \n",
       "a                                          -1.0   \n",
       "b                                           1.0   \n",
       "\n",
       "variable  pressure__linear_trend__attr_\"slope\"  \\\n",
       "id                                               \n",
       "a                                         -2.0   \n",
       "b                                          8.0   \n",
       "\n",
       "variable  pressure__linear_trend__attr_\"stderr\"  \\\n",
       "id                                                \n",
       "a                                           0.0   \n",
       "b                                           0.0   \n",
       "\n",
       "variable  temperature__linear_trend__attr_\"intercept\"  \\\n",
       "id                                                      \n",
       "a                                                 1.0   \n",
       "b                                                 3.0   \n",
       "\n",
       "variable  temperature__linear_trend__attr_\"pvalue\"  \\\n",
       "id                                                   \n",
       "a                                              0.0   \n",
       "b                                              0.0   \n",
       "\n",
       "variable  temperature__linear_trend__attr_\"rvalue\"  \\\n",
       "id                                                   \n",
       "a                                              1.0   \n",
       "b                                             -1.0   \n",
       "\n",
       "variable  temperature__linear_trend__attr_\"slope\"  \\\n",
       "id                                                  \n",
       "a                                             1.0   \n",
       "b                                            -2.0   \n",
       "\n",
       "variable  temperature__linear_trend__attr_\"stderr\"  \n",
       "id                                                  \n",
       "a                                              0.0  \n",
       "b                                              0.0  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_tsfresh = extract_features(df, column_id=\"id\", column_value='value', column_kind='kind',\n",
    "                             default_fc_parameters=settings_regular)\n",
    "X_tsfresh.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Writing your own time-based feature calculators\n",
    "\n",
    "Writing your own time-based feature calculators is no different from usual. Only two new properties must be set using the `@set_property` decorator:\n",
    "\n",
    "1) `@set_property(\"input\", \"pd.Series\")` tells the function that the input of the function is a `pd.Series` rather than a numpy array. This allows the index to be used.\n",
    "2) `@set_property(\"index_type\", pd.DatetimeIndex)` tells the function that the input is a DatetimeIndex, allowing it to perform calculations based on time datatypes.\n",
    "\n",
    "For example, if we want to write a function that calculates the time between the first and last measurement, it could look something like this:\n",
    "\n",
    "```python\n",
    "@set_property(\"input\", \"pd.Series\")\n",
    "@set_property(\"index_type\", pd.DatetimeIndex)\n",
    "def timespan(x, param):\n",
    "    ix = x.index\n",
    "\n",
    "    # Get differences between the last timestamp and the first timestamp in seconds, then convert to hours.\n",
    "    times_seconds = (ix[-1] - ix[0]).total_seconds()\n",
    "    return times_seconds / float(3600)\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
